{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "### DictVectorizer",
   "id": "890763a9e3e2dfb8"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "DictVectorizer主要用于处理由字典（或类似字典的结构，如 pandas DataFrame）表示的数据。它可以将字典数据（通常是稀疏的）转换为适合机器学习算法处理的特征矩阵。它将字典类型的输入（如 Python 字典、pandas DataFrame 或列表等）转换为一个数值矩阵，该矩阵的列表示特征，行表示样本。",
   "id": "aecf164d4bb38db4"
  },
  {
   "cell_type": "code",
   "id": "initial_id",
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2025-01-09T14:36:57.548575Z",
     "start_time": "2025-01-09T14:36:56.433197Z"
    }
   },
   "source": "from sklearn.feature_extraction import DictVectorizer",
   "outputs": [],
   "execution_count": 2
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T14:36:56.432195Z",
     "start_time": "2025-01-09T14:36:56.427084Z"
    }
   },
   "cell_type": "code",
   "source": [
    "#创建字典\n",
    "data = [{'city': 'New York', 'temperature': 25},\n",
    "    {'city': 'San Francisco', 'temperature': 18},\n",
    "    {'city': 'Austin', 'temperature': 30}]"
   ],
   "id": "b798d1b22a072e9f",
   "outputs": [],
   "execution_count": 1
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "DictVectorizer 支持稀疏和密集矩阵格式的输出。默认输出稀疏矩阵。",
   "id": "1f36e56d5de13500"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T14:46:36.214423Z",
     "start_time": "2025-01-09T14:46:36.210622Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# dtype参数决定输出矩阵的数据类型，默认为 float64。\n",
    "vec = DictVectorizer()\n",
    "X = vec.fit_transform(data)\n",
    "print(X)"
   ],
   "id": "6b2e15dccc48d677",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Compressed Sparse Row sparse matrix of dtype 'float64'\n",
      "\twith 6 stored elements and shape (3, 4)>\n",
      "  Coords\tValues\n",
      "  (0, 1)\t1.0\n",
      "  (0, 3)\t25.0\n",
      "  (1, 2)\t1.0\n",
      "  (1, 3)\t18.0\n",
      "  (2, 0)\t1.0\n",
      "  (2, 3)\t30.0\n"
     ]
    }
   ],
   "execution_count": 7
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "通过设置 sparse=False 参数，可以让输出结果为密集矩阵格式",
   "id": "c627444df9f65b5a"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T14:37:00.440334Z",
     "start_time": "2025-01-09T14:37:00.433911Z"
    }
   },
   "cell_type": "code",
   "source": [
    "vec = DictVectorizer(sparse=False)\n",
    "X = vec.fit_transform(data)\n",
    "print(X)"
   ],
   "id": "381223ce72b52635",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  1.  0. 25.]\n",
      " [ 0.  0.  1. 18.]\n",
      " [ 1.  0.  0. 30.]]\n"
     ]
    }
   ],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "在这个输出中，每个词典字段（city 和 temperature）被转换为矩阵的列，其中城市名称被独热码（one-hot encoding），temperature 则直接作为数值特征。",
   "id": "e686aaf0439163c8"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "",
   "id": "5597a4d2504a599e"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T14:44:55.439882Z",
     "start_time": "2025-01-09T14:44:55.435192Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 查看由 DictVectorizer创建的列名\n",
    "print(vec.get_feature_names_out())"
   ],
   "id": "3a2df21d1c120dcd",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['city=Austin' 'city=New York' 'city=San Francisco' 'temperature']\n"
     ]
    }
   ],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "### CountVectorizer",
   "id": "cbab4e23f9863732"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "CountVectorizer用于将文本数据转换为词频矩阵（bag-of-words 表示）。它通过对文本数据进行分词处理，生成一个表示每个单词在文本中出现次数的特征矩阵，是文本处理中的一个常见步骤。\n",
    "\n",
    "词袋模型（Bag of Words）：CountVectorizer 将文本转化为特征矩阵，其中每一列代表一个单词，每一行代表一个文档（文本）。矩阵中的值表示该单词在对应文档中出现的次数。\n",
    "\n",
    "特征提取：用于从文本数据中提取特征，通常用来为文本分类、聚类等任务生成数值特征。\n",
    "\n",
    "相关参数：\n",
    "\n",
    "stop_words: 可选，去除常见的停用词（如 \"the\", \"is\" 等）。默认值是 None。可以设置为 'english' 来移除英文常用停用词，也可以传入一个自定义的停用词列表。\n",
    "\n",
    "max_features: 只保留频率最高的 n 个词。\n",
    "\n",
    "ngram_range: 设置生成的词组（n-gram）的范围。例如 (1, 2) 会生成 unigram 和 bigram（即单个单词和两两单词组合）特征。\n",
    "\n",
    "lowercase: 是否将所有文本转为小写。"
   ],
   "id": "fe26e2bd767a3624"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:03:43.305829Z",
     "start_time": "2025-01-09T15:03:43.301987Z"
    }
   },
   "cell_type": "code",
   "source": "from sklearn.feature_extraction.text import CountVectorizer",
   "id": "28dcbcabf9f7e93a",
   "outputs": [],
   "execution_count": 13
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:03:45.101499Z",
     "start_time": "2025-01-09T15:03:45.098628Z"
    }
   },
   "cell_type": "code",
   "source": [
    "documents = [\"I love programming in Python\",\n",
    "    \"Python is great for machine learning\",\n",
    "    \"I enjoy learning new things in Python\"]"
   ],
   "id": "4e5d84d3c0abfd84",
   "outputs": [],
   "execution_count": 14
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "使用 CountVectorizer 将文本转换为词频矩阵，X 是一个稀疏矩阵，包含每个文档中每个词的出现次数。",
   "id": "dde8c93dfa88c460"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:15:38.649345Z",
     "start_time": "2025-01-09T15:15:38.644344Z"
    }
   },
   "cell_type": "code",
   "source": [
    "#将文本转换为词频矩阵\n",
    "vectorizer = CountVectorizer(stop_words='english')# 去除英语常用停用词\n",
    "X = vectorizer.fit_transform(documents)# x为稀疏矩阵三元组格式\n",
    "print(X)"
   ],
   "id": "311fb02455cac84",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Compressed Sparse Row sparse matrix of dtype 'int64'\n",
      "\twith 12 stored elements and shape (3, 9)>\n",
      "  Coords\tValues\n",
      "  (0, 3)\t1\n",
      "  (0, 6)\t1\n",
      "  (0, 7)\t1\n",
      "  (1, 7)\t1\n",
      "  (1, 1)\t1\n",
      "  (1, 4)\t1\n",
      "  (1, 2)\t1\n",
      "  (2, 7)\t1\n",
      "  (2, 2)\t1\n",
      "  (2, 0)\t1\n",
      "  (2, 5)\t1\n",
      "  (2, 8)\t1\n"
     ]
    }
   ],
   "execution_count": 17
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:16:33.590319Z",
     "start_time": "2025-01-09T15:16:33.587293Z"
    }
   },
   "cell_type": "code",
   "source": "print(X.toarray())# 将x转换为数组格式",
   "id": "4fddf96b8a69b805",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 1 0 0 1 1 0]\n",
      " [0 1 1 0 1 0 0 1 0]\n",
      " [1 0 1 0 0 1 0 1 1]]\n"
     ]
    }
   ],
   "execution_count": 19
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "通过 get_feature_names_out() 可以查看生成的词汇表，即每一列对应的单词。\n",
    "\n",
    "词汇表的类型为numpy.ndarray"
   ],
   "id": "7a71bb7d798b6fe5"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:19:03.541811Z",
     "start_time": "2025-01-09T15:19:03.538947Z"
    }
   },
   "cell_type": "code",
   "source": [
    "print(vectorizer.get_feature_names_out())# 打印词汇表\n",
    "print(type(vectorizer.get_feature_names_out()))"
   ],
   "id": "2b45e8c235c605e2",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['enjoy' 'great' 'learning' 'love' 'machine' 'new' 'programming' 'python'\n",
      " 'things']\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "execution_count": 21
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "### TfidfVectorizer",
   "id": "e92ddd8d3348be91"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "TfidfVectorizer用于将文本数据转换为 TF-IDF（词频-逆文档频率）特征矩阵。它是 CountVectorizer 的改进版本，解决了单纯依赖词频的缺陷，能够减少高频词（如“the”, “is”, “and”等常用词）对模型的影响，提升模型对重要单词的敏感度。\n",
    "\n",
    "TF(t,d)= 词t在文档d中出现的次数 / 文档d中的总词数\n",
    "\n",
    "\n",
    "IDF(t)=log[ N / (包含单词t的文档数+1) ]\n",
    "\n",
    "其中，N 是语料库中的文档总数，包含单词t的文档数 表示含有单词 t 的文档数。\n",
    "\n",
    "TF-IDF 是这两个值的乘积：\n",
    "\n",
    "TF-IDF(t,d)=TF(t,d)×IDF(t)\n",
    "\n",
    "TF-IDF 的值越高，说明单词 t 对文档 d 的重要性越大。\n",
    "\n",
    "相关参数：\n",
    "\n",
    "stop_words：用来指定是否去除停用词，可以设置为 'english'（去除英语常用停用词）或传入自定义的停用词列表。\n",
    "\n",
    "max_features：指定返回的特征矩阵的最大特征数。只保留频率最高的 n 个词。默认是 None。\n",
    "\n",
    "ngram_range：指定生成的 n-gram 的范围（如 (1, 2) 会生成 unigram 和 bigram）。\n",
    "\n",
    "use_idf：是否使用 IDF（默认是 True，启用 IDF），如果设置为 False，即为仅使用词频（类似 CountVectorizer）。\n",
    "\n",
    "smooth_idf：启用平滑 IDF，默认为 True，可防止某些词的 IDF 计算为负值。\n",
    "\n",
    "sublinear_tf：是否使用对数尺度的词频（如 log(1 + tf)），默认是 False。"
   ],
   "id": "96279ea6c3edb496"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:29:04.582412Z",
     "start_time": "2025-01-09T15:29:04.577758Z"
    }
   },
   "cell_type": "code",
   "source": "from sklearn.feature_extraction.text import TfidfVectorizer",
   "id": "2cf71ef0e1d9af9",
   "outputs": [],
   "execution_count": 22
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:43:37.795829Z",
     "start_time": "2025-01-09T15:43:37.792471Z"
    }
   },
   "cell_type": "code",
   "source": [
    "documents = [\"I love programming in Python\",\n",
    "    \"Python is great for machine learning\",\n",
    "    \"I enjoy learning new things in Python\"]"
   ],
   "id": "9d58e3c13c3fdf61",
   "outputs": [],
   "execution_count": 30
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "使用 TfidfVectorizer 将其转换为 TF-IDF 特征矩阵 X， X 是一个稀疏矩阵，表示每个文档的 TF-IDF 特征。",
   "id": "6343a319475068ac"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:49:54.217860Z",
     "start_time": "2025-01-09T15:49:54.212998Z"
    }
   },
   "cell_type": "code",
   "source": [
    "vectorizer = TfidfVectorizer(stop_words='english')# 去除英语常用停用词\n",
    "X = vectorizer.fit_transform(documents)\n",
    "print(X)\n",
    "print(type(X))# x的类型是scipy.sparse._csr.csr_matrix"
   ],
   "id": "21ebc8034ce3cc5c",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Compressed Sparse Row sparse matrix of dtype 'float64'\n",
      "\twith 12 stored elements and shape (3, 9)>\n",
      "  Coords\tValues\n",
      "  (0, 3)\t0.652490884512534\n",
      "  (0, 6)\t0.652490884512534\n",
      "  (0, 7)\t0.3853716274664007\n",
      "  (1, 7)\t0.34520501686496574\n",
      "  (1, 1)\t0.5844829010200651\n",
      "  (1, 4)\t0.5844829010200651\n",
      "  (1, 2)\t0.444514311537431\n",
      "  (2, 7)\t0.2980315863446099\n",
      "  (2, 2)\t0.3837699307603192\n",
      "  (2, 0)\t0.5046113401371842\n",
      "  (2, 5)\t0.5046113401371842\n",
      "  (2, 8)\t0.5046113401371842\n",
      "<class 'scipy.sparse._csr.csr_matrix'>\n"
     ]
    }
   ],
   "execution_count": 36
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:51:29.319949Z",
     "start_time": "2025-01-09T15:51:29.316403Z"
    }
   },
   "cell_type": "code",
   "source": "print(X.toarray())# 将x转换为数组格式",
   "id": "38c8b4330fb2d46c",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.         0.         0.         0.65249088 0.         0.\n",
      "  0.65249088 0.38537163 0.        ]\n",
      " [0.         0.5844829  0.44451431 0.         0.5844829  0.\n",
      "  0.         0.34520502 0.        ]\n",
      " [0.50461134 0.         0.38376993 0.         0.         0.50461134\n",
      "  0.         0.29803159 0.50461134]]\n"
     ]
    }
   ],
   "execution_count": 39
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "通过 get_feature_names_out() 可以查看生成的词汇表，即每一列对应的单词。\n",
    "\n",
    "词汇表的类型为numpy.ndarray"
   ],
   "id": "8d77aa05bd14b17d"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-01-09T15:51:47.452036Z",
     "start_time": "2025-01-09T15:51:47.448176Z"
    }
   },
   "cell_type": "code",
   "source": [
    "print(vectorizer.get_feature_names_out())\n",
    "print(type(vectorizer.get_feature_names_out()))"
   ],
   "id": "c92e0b8eaf33a021",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['enjoy' 'great' 'learning' 'love' 'machine' 'new' 'programming' 'python'\n",
      " 'things']\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "execution_count": 40
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
